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ABSTRACT This evaluation of cognitive change in Head Start 

children focused on changes in performance as opposed to changes in 
competence; specifically,' that Binet test performance ^ 

function of experience with Binet examiners. The study mvolvea 93 
children assigned to four groups v?ho were tested for IQ^gains during 

a 6-week Head Start program in summer, 1968. Group 

sex was approximately equal; distribution by race was also equal in 
Groups 1, II, and III. Group TV had only black children with pretest 
?Q that was meaningfully lower than that of the other groups. Groups 
were given the initial Stanf ord-Binet either at home or during the 
first, or second week of school. A-t the end of the program^ ^ P 

was again tested. A preliminary analysis of 

pretest IQ scores and the mean posttest scores showed that the four 
groups were essentially similar at the beginning and end ot the 

results iPdicape that, despite the possible varaataons 
in ?S”ne?s and programs and In pretest IQ, overall differences among 
the groups were Smah and random indicating that test performance dad 
not improve. Magnitude of change is also dascussed an - C 

available in hard copy due to marginal legibility of 
document^. (MK) 
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CiuM-lGES IN S'UUPvOIi*^-BIWET IQ. 



PEUPOR’.lAi\CE VS COilPETENCE^ 



Williou J. Keyor and Byron Escland 



The evaluation of chansea in children's behavior as a result of experiences 

ad Start classes, or ocher compensatory education programs, has typically 
✓ 

ved a pre-test post-test design. Thus a single measure such as the 
ord-hinet test, or a battery of generally correlated cognitive measures, 
,dministered prior to the treatment experience and depending on the 
ion of the program (six v/eek programs vs full year programs) the same 



:ry is readministered and differences in performance are assessed, 
tpriacely , the _t_—tesc is used co evaluate the statistical significance 
ve change score. Since this statistical test is particularily sensitive 
lange , the results of the analyses arc commonly statistically significant. 

An important question of both theoretical and j actical concern is 
aeaning that may be aesaoribed to such findings. ^ince the original 
jlation of this study, Zigler and Butterfield (I960) published a paper 
sing on the same Issues of concern here, (indeed the research di?.sign is 
similar to the one used in this study). According to their theoretical 
ysis, with which we are in fundamental agreement, IQ changes may reflect: 
Changes in formal cognitive processes, that is, changce in the level of 
itivc functioning which we are labeling ’’competence . (2) Changes in 

raacion levels, that is, changes that reflect increased knowledge but 
changes in level of cognitive functioning; and (3) ’’^lotivational factors' 
;h, fxom our viewpoint, reflect changes in the willingness to attempt 
is as opposed to passive nonresponding. 

Aside from the Zigler and Butterfield study which provides suppore for 
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clui ciocivacional inctirp rotation, chare ore no studies which have coma to our 
attention focusinij oi\ tho nerfonnanco vo .coiT^pcccncc interpretation. Clearly 
Ziglor nl . take tne position ^hat changes In motivation generate changes 
in pertonaance level, but not nocesGarily in cognitive levels. It is our 
hypoch.’si^ that the commonly reported IQ gains of between four and eight 
IQ points reflect motivational changes. It is our further hypothesi's that 7 
these motivational changes are reflected directly in performance rather than 
in competence changes. 

The gencr- 1 design of this study involves four groups of children who 
received a pre- and a post-test at differem', points during a six week summer 
Hoad Start Program; this phase of the research design relates to the motiv- 
ational interpretation. T.hc- second component of this project involves an 
item ■hcflly.e.is of the Scanford-Binets for each of the 100 children in the 
sample. These analyses will focus on the nature of the pre-test items 
passed and the post-test items passed and will examine the nature of the 
post-test items passed. In performing this latter it is our 

assumption that certain of the Binet items appearing at tho six-year level 
are indicative of higher cognitive levels and thus may reflect changes in 
level. This first report is related to the Impact of the Initial testing 
experience on changes in motivation. 



i'iETHOD 

Subjects The children in this study were enrolled in a six week head 
Start Program during the Summer. 1968. All the children resided in what 
is generally Icnown as the ''inner city*'. Of the original sample of 100 
children t two children were unavailable for pre-testing and an additional 
five children were not available, largely because they moved away, for the 
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post -tcsciu^j . XauIo 1 dhows thts Wa ar.d CA.u by xa.cn, and treatncrit 

I 

group, 

Uniorcunaculy , it was not poaoibla to randomly asuign children to each 
of the treatment groups. A preliminary analysis of the data indicated, 
however, that the differences bet ?een the groups oa pre-test IQ are not 
statistically significant (JF ■“ 2.A;- df * 3 & 9A : 2 . >*05). The distribution 
by sex for all four groups is approximately equal as is the distribution 
for race in Groups I, II, and III. Group IV is comprised entirely of Jlegro 
children and despite the failure to attain statistical significance it 
would appear that the Croup IV pre-test IQ is perhaps meaningfully lower 
than, that of the other groups. The significance of these sampling differences 
will be discussed later in this paper in context with the data. 

Procedure There were a total of four groups which varied in teirms of when 
they were given the initial Stanford— Binet during the six week head Start 
Program. Group I was initially tested at home duriiig the week prior to 
the beginning of the head Start Program. It was thought that 'irhan*; 
administering the test in famiULar surroundings ' might minimize the impact 
of having to work with a strange and somewhat demanding adult. This group 
was post-tested during the last week of the Head Starr. Program. Group II 
was administered the Binet during the first week of the Head Start Program 
and during the last, or sixth week. This group is considered the standard 
or control group. Group III was administered the Stanford— Binet during the 
second week and again during the sixth week. It was hypothesized that 
this group might benefit from one week of encounters with strangers and 
would thus show higher pre-teat performance and conceivably a lower change 
score. Group IV was administered the Stanford-Binat during the first week. 
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Che second weelt,' and the sixth week. This is che critical croup in our 
desiijr, because it can show the effects of experience with examiners (week one 
vs week two) and can further indicate if there are meaningful practice effects 
from repeated testing (week two vs week six) . 

All children were administered the full-length Staaford-Binet by 
examiners experienced in working with Head Start children. Standard testing 
procedures were enployedi that is » no specific efforts xtara made to maximize 
the children's performance. 

The Nursery School Programs in which the children were enrolled can 
mv'>st appropriately be described as traditional in approach. This implies 
that clva children were provided with standard -nursery school equipment, such 

f 

as blocks, dress-up corners, and other toys, and a prograra involving such 
things reading stories, coloring and pasting, and field trips. Bach 
classroom was comprised of a head teacher trained in early childhood 
education and a teacher-aide who v^cre mainly from the neighborhood. (These 
comments on the nature of the program are derived entirely from general 
Impressions since we made no effort to document program content in detail.) 

Results 

A preliminary analysis of variance of the mean pre-test 10 scores and 
the mean post-test IQ scores Indicates that the four groups are essentially 
similar at Che beginning and at the end of the programs (.? “ 2,4; df ..3 • 
d 94, .05; ^ “ 2«i0,df^ = 3 d 90 2.” ^*05, pre- and post-tests, respect- 

ively) . These results indicate chat despite the possible variations in 
teachers ar»d programs and despite the variation in pre-test IQ, the . 

overall differences among the groups at the conclusion of Che six-week program 

O 

small and random. 
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Noxt; wo exuainvid chc ma£r.icudo of chanc-;. within each of the four treat- 
sent jjroups. Table 2 suamarizea the meana and SDs of the pre- and post-test 
Bcans and chv 2 ncan chause scores for each of the srouns. A series of catched 
t testa to determino the sisiUflcance of the chansc scores were run for 
each group. Group I. ^ - 4.2 df - 22 . ^ . 01 : Group II, t » 2.8 d_f - 

24. Ji <r.01: Group III, t - 3.9, - 24 . p. “ 's .01. Group IV analysis 

involved a within groups analysis of variance conparing Week 1 vs Week 2 
vs Weuk 6. F » 6.4. ^ - 2962: f <.01). Jiultiple comparisons were run 

between Week 1 and Week 2 and Week 2 and Week 6 (t « 3.6; ^ « 22. 2. < -01 

and ^ .2. « 22, » ^.05 respectively). These results Indicate that 

Groups 1, II-p and III, the average gain in IQ score is statistically 
sisnxficant and is of h magnitude commensurate with that frequently reported 
in otner studies (see the. control group gains in the Ziglar and Butterfield 
study, for example). It would appear, therefore, that nothing unusual is 
occurring i.n our sample of children or classroom teachers. Of oarticular 

C 

pertinence to this study, however, is the fact that in Group IV the change 
in IQ between Week 1 and Week 2 is statistically significant and of the 
same magnitude ar. the other groups. The change between Week 2 and Week 6 
IS not or.ly not statistically significant but the children actually showed a 
‘very slight drop in performance. These results are consistent with our 
.hypothesis, and that proposed by Zigler and Butterfield (1960), that the- ' 
typical gains reported in -IQ may well be a function of experience with 
•examiners as contrasted to the experiences gained in Head Start classrooms. 

The next phase of the analyses compared the difference scores for each 
os the four groups. In the first batween-groups analysis, the difference 
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U<2tans and Standard Deviations 
Pra-TasCy Post-Tost, Changa Scores 
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X and V/aak 2 and 



licora includod for Giroup IV was che difforonco batwaar. 
the second bacwacndgroups analyses of variance involved the difforanccs 
between Week 2 and Week 6, The results of these analyses were; .44, 

dt -3d S9, “ ^.05 and ?. « 3.99, -3d G9, n_ » <1 .05, respectively. 

The results of the first analyses of variance sus^eat that the effects of 
costing the children at homo, delayins the first administration of the Binut ' 
until the second week, and the effect of post-testinej after only one week, 
had no significant afflicts on change scores. These results can tentatively 
be interpreted as meanins that familiarity of surroundings and one week of 
experience with adults in a school setting does not particularly influence 
pre-test performance. This data further indicate that extended (5 week) 
expuricncas in a program does not effect changes. This conclusion is further 
supported by the results of the analyses of variance of tha pre-test 
scores, previously reported, in which there were no statistically significant 
differences between the four groups. 

Of particular interest is the result of the second analyses of variance 
in which the change score for Group IV involved the difference between 
performance on the tost during Week 2 and during Weak ,6. In considering 
these results it should be noted that the mean difference for Group II and 
the mean difference for Group IV, taking the difference in Group IV as Week 
1,^ 'Wav:k 6; are essentially identical. A comparison of the change scor’es 
for Group III (tested Week 2 and Week 6) with the comparablw period oJ 
time for Group IV reveals a statistically significant difference 2.67, 

df - 44. 2 . “ ^•01). Differences butween change scores for Group I vs 
Croup II and Group II vs Group III are not statistically significant 
- .85, “ 4Si £, « 



>>.05; and t. 



03; dX “ ^5’ 2 . “ respectively) 



Tlvi rsisulca o'f chciia analydcjG suegaac rather strongly that the observed 
uiagaicuJo’ of chav>se in IQ performanew in Groups I through III occurs as a 
result of thu initial uxavaination and not as a result of time in the IVoad 
Scare situation. 

In addition to the analyses concuriwd with the aajor issues in this 
study, addicioi\al .analyses were perforriied which may be of some intarost. 

It was possible, for example, to examine sex and race differences on both 
Che pre- test and, post- teat scores as well as the raasnitude of chaiige scores. 

A two by two analysis of variance of the pre— test scores inelcated tr.at tae 
-sex difference is not statistically significant (£ “ 3.4 1 96, v,^.03) 

that the race difference is not statistically significant (F « 3.0, d f. - 1 & 

2 . >.03); and sex x race interaction is not statistically significant 
(? » 1.2; d£ 1 6 96i 2 ” ’ Performance on the post-test indicated 

only a statistically significant sex difference (X* S.li d_^ 1 4 91; 2 “ 

.01). Inspection of the pro— test data indicates that the direction of 
th.^ sex difference is in favor of the males; whereas tha direction of the 
race difference is in favor of the white children. The same directional 
dif f<srences occur for the post— test but these differences achieve statistical 
sicr.ificiinca only in the case of sex. In view of the fact that the recruiting 
practices employed in making up the various groups are vu^knovm, it is not 
particularly -clear why the male subjects, and this was true for both racial 

"V - ^ 

groups, performed bettor on the Stnnford-Sinet than the females. 

A frequently reported finding is a correlation between the pre-test 
score and the magnitude of the change score. Typically, this correlation is 



In a negative direction indicating that children with lower initio? scores 
O tend to gain somewhat more on the post-test, la the context of this study. 
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chii corL'olation could ba inter pro tod a a indicatinij that childrcri 

lower pro-coat scoroa ai*c t;oro advoracly affected by their initial intoraccion 
with the oxaniuar and, conversely, benefit noro frora .the initial exai-.-.natxo*. 
oxperienco than children with hlpher pre-test scores. In this otudy none 
of the correlations botv/eon pre— test performance and chanji^ score wore 
statistically significant. This result eugs^^sts that resardless of initial 
cost perforrtance ^ the children benefited in a uniform fashion xron t.«c 
original oxpericncvi with the examiuer. 



D i s c s i o n 

Thu purposci o£ this phase of the project concerned v;ith the evaluation 
of cognitive chanse in Head Start childr-en focused on changiis in perforaance 
as opposed to chanQds in coapetence. liorc specifically, this study 
examined the hypothesis that Binet test performance improves as a function 
of experience with Binet examiners. ^»s part of the overall design, this 
study examined the effects of pra-testing at home in a presumably familiar 
environment where the childrei'. would perhaps be more responsive and also 
examined the effects of delaying the initial examination for a week, giving 
the children the experience with strange adults. The major outcome of the 
study was chat as much gain in performance occurred in one week as occurred 
over a period of at least five weeks. The effect of testing at homu and 
“of delaying the initial examinatian for one weuk were not only not 
statistically significant but did not influence average performance to any 
degree whatsoever. 

Precise and unequivocal interpretations of the data derived from, this 
study are impossible for at least three reasons ; (a) the children were not 

O randomly placed in the different treatment groups. (b)‘ there was no .control 

.EJJC . 
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j^roup thac had bean prc-tuated and poiit-tesuecl v^iUhout iatirv'inir.^ 
exporiene^j (c) che duracion of tiivj pro.'.-.rara v/as ot*ly for a oerlod of six 
w^oka. In uhc case of the firac liiaitationj at least three supnortivo 
ar^uraeuts can be vuade; (1) deapite the lov:ar pre-test IQ cf Group IV,. all 
four ijroups evidenced essentially -the oaae r.aount of sain and this gain 
(4.6 IQ points) is essentially sivailar to that reported by Zigler and iuttcr- 
field (1963.^ (2) the correlation between pre -test score and chanse score is 
not statistically significant for any of the groupS'-'-suggestinr that pre-test 
variation did not influence tiagaitude of change. (3) it is unclear v/hat 
subject characteristics aight exist that would allow for a 4,6 gain in IQ 
between the first and second testing „ but a -loss of .3 IQ points betv/een 
the second and third testing. It is conceivable y of course. ^ that sor.e 
variables are operating and this night well be worth exaaining in a truly 
ranooitm.je design* 

kith respect to the third limitation of this study, naaely the six- 
week duration, it should be noted that the mean change found in this study 
is cotiri.jnsurate with the outcome of full- year Head Start programs. Fror?. 
the basic viewpoint adopt j.d in this study, this outcona is not surprisin? . 

As noted earlier^ the frequently reported significant negative correlation 
bctvraen pre-C'*;at porfonaance and change score, ougg'-sbs that in those studies 

reporting lergdr mean gains it is conceivable that the subjects had lovjer 

a 

initial test scores. It is /further contention, hovrever, that the magnitude 
of IQ change may be a relatively unimportant variable. Thus* even an IQ change 
of as much as 10 or 12 points may not, depending upon the initial test score, 
signify any meaningful change in level of cognitive functioning. The 
Q ' igree to which IQ gain does reflect shifts in cog'n?' tivc livol is currently 
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b%iiu 2 , cxirainud in our Inboratory. 

A coiaTciunt or two soe;n3 in ordor on tho 3ar;.'Jli of chilurur. initiclly 
c;i3tod at houio. Aocall that it v:aa our hyoochosis that thosa children vrould 
perhaps show a Baallor gain score bccausa their pr^-tost acoras would not 
havu bean adversely influenced by strange surroundings. Clearly, this did 
not occur, indeed, it almost appears as if these children were adversely 
affected by having their initial ex?::^in.r ■ ::.on at home. The jurior author did 
the initial home exaiaination and ac rdi'i g to his reports there were 
substantial dis - ractions at home in t. :rr. of general noise lc^■el deriving 

botr. from adults and from other children., Thus, it 'would appear that the 

} 

procedure of examining children at hone is not feasible. 

wir.ally, these data cannot be interpreted ns indicating anything 
particularly negative about the .four ticad Start clvnssas that ware employed. 
Cur observations indicated that these classrooms v^ere not atypical of other 
liead Start classes in our experience and that the teachers were both well- 
trained anc coumitted to the program. It should also be noted that we have 
little information to datarmine why some children shov;ed larger gains than 
others, yet this variation in change score certa.inly suggests that there 
were attributes in, each of these classrooms that positively infl-eanced at 
least some children. If anything, the results of this study in conjunction 
with those reported by Ziglcr and Butterfield suggest that a considerable 
amount of basic work is needed before we will gain an understandinS" of' ’ 
cognrtive development and thus be in a position to develop better programs. 
One implication of these data, for other investigators, is to consider the 
magnitude of IQ change scor-es in their ov/n studies in terms of the amount 
of IQ gain to he anticipated on the basis of experisnee with examiners. 
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Purl;a?s tV.c inpoirtcu'.t ivapliccition. ■^lowiV^;r; 

scraca-iy fr-r’.u r^iporcini: chanj,:a scoria to a stra!-:,; 
iav^isci^ato cUa charactoriatics of children "as ta 
chavactotiatics of tha prograao'— an of Con stated 



ia the naad for a shift in 
in wl'.ich v;a int^n'.sivaiy 
y ,-tetact tha 

’ccriva • at secahow has 



not yat been implaa<4ntoJ. 












